Autonomous vehicle vision system

ABSTRACT

In accordance with one aspect of the invention, there is provided a method of autonomously operating a vehicle. The method provides at least two cameras in operable communication with the vehicle for providing substantially similar views relative to the vehicle. The at least two cameras receive information relating to the views. The method also provides a laser in operable communication with the vehicle for selectively shining a single discrete mark on at least a portion of the views provided by the at least two cameras. Further, the method determines whether the information received by the at least two cameras is ambiguous regarding the views. The method activates the laser on at least a portion of the views based on whether the information received by the at least two cameras is ambiguous.

PRIORITY INFORMATION

The present application is a continuation of U.S. patent applicationSer. No. 13/906,549, filed May 31, 2013, the contents of which isincorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates generally to vision systems for autonomousvehicles such as a robot or the like and, in at least one embodiment, tosuch vision systems that accurately determine the distance a vehicle maybe from another item/object, or whether the item/object is in the fieldof the vehicle.

2. Introduction

Autonomous vehicles such as robots or other vehicles typically need theability to recognize and steer around objects that may be in its path.To assist in accomplishing such tasks, various types of cameras or thelike can be used.

When an object has depth or texture to it, cameras typically are able todetermine the distance between the robot and the object and readilyidentify and/or steer around it as desired. When an object has little ifany depth to it, such as a flat wall, for example, recognizing theobject can present difficulties.

SUMMARY

Additional features and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. Thefeatures and advantages of the invention may be realized and obtained bymeans of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present inventionwill become more fully apparent from the following description andappended claims, or may be learned by the practice of the invention asset forth herein.

In accordance with one embodiment of the invention, a method ofautonomously operating a vehicle provides at least two cameras inoperable communication with the vehicle for providing substantiallysimilar views relative to the vehicle. The at least two cameras receiveinformation relating to the views. The method also provides a laser inoperable communication with the vehicle for shining a single discretemark on at least a portion of the views provided by the at least twocameras. Further, the method determines whether the information receivedby the at least two cameras is ambiguous regarding the views. The methodactivates the laser on at least a portion of the views based on whetherthe information received by the at least two cameras is ambiguous.

In some embodiments, the method provides the at least two cameras forproviding overlapping views relative to the vehicle. The views relativeto the vehicle may overlap by at least a threshold percentage. The viewsrelative to the vehicle may have fields of view that overlap by at leasta threshold angle. In some embodiments, the laser shines a dot, asquare, a diamond, or an irregular shape. The discrete mark may have acurvature, a straight edge, and/or both. In many embodiments, thediscrete mark has a solid interior.

The method may determine whether the information relating to the viewsis ambiguous in any of a number of ways. For example, the informationmay be ambiguous if a result of a formula (when inputs based on theinformation in the substantially similar views are used in the formula)exceeds a predetermined threshold. In another example, the informationmay be ambiguous if the distance to an object in the views cannot belogically determined.

In some embodiments, the method activates the laser at least or onlywhen the information is ambiguous. The method may not activate the laserwhen the information is not ambiguous.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the invention can be obtained, a moreparticular description of the invention briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only exemplary embodiments of the invention and are not thereforeto be considered to be limiting of its scope, the invention will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 is a block diagram of an exemplary vehicle equipped with aprocessing unit coupled to two cameras and a laser.

FIG. 2 is a block diagram of an exemplary vehicle with cameras facingthe same direction and are sufficiently close such that the camerascapture substantially similar views of the vehicle's environment.

FIG. 3 is a block diagram of an exemplary vehicle with cameras facingthe same direction and are positioned behind one another such that thefield of view of one camera encompasses the field of view of the othercamera, thus capturing substantially similar views of the vehicle'senvironment.

FIG. 4 is a block diagram of an exemplary vehicle with cameras angledtowards one another such that a portion of their field of viewsintersect, thus capturing substantially similar views of the vehicle'senvironment.

FIGS. 5-10 depict exemplary discrete marks that the laser 120 couldproject.

FIG. 11 is an exemplary flow diagram of a method of autonomouslyoperating a vehicle in accordance with illustrative embodiments of theinvention.

DETAILED DESCRIPTION

Embodiments of the present invention will be described below withreference to the accompanying drawings. It should be understood that thefollowing description is intended to describe exemplary embodiments ofthe invention, and not to limit the invention.

Autonomous vehicles, such as robots, can be deployed to perform tasks inlocations that are remote and/or risky. In various examples, vehiclesmay be used to clear debris from a site impacted by a natural disaster,such as a hurricane, earthquake, or tornado. Vehicles may also bedeployed to retrieve the bodies of injured soldiers from a war zone.Further, vehicles may clean up or remove dangerous substances, such asradioactive material or hazardous biological waste, from sites.

In many situations, vehicles are sent to locations with little orunreliable infrastructure. As a result, the vehicles may have onlyintermittent access to energy sources, such as electricity grids orfueling stations. Further, the vehicles' tasks may require them toremain on-site for extended periods of time. Thus, it is advantageousfor vehicles to use their energy efficiently so that they may accomplishas much as possible in the field before personnel must retrieve thevehicles for refueling, maintenance, and/or repair.

The ability for a robot to move faster through its environment is alsoone of many aspects of the present invention that directly relates toenergy savings. The more a robot is slowed down during operation by anabnormality in its vision system the more energy is used for the robotto complete its task. Additionally, constantly using a laser fordistance or other article determination in conjunction with cameras hasthe potential to use more energy since integration of the two systemscan take time and energy and may create more anomalies than otherwisemay occur when merely using cameras.

Vehicles often use cameras to recognize objects that may be in theirpaths. In some situations, by processing image data from the cameras, avehicle is able to determine the distance to an object in its path andsteer itself around the object accordingly, or take some other action.However, in some situations, a vehicle cannot determine a distance basedsolely on the image data. To overcome this problem, in illustrativeembodiments, the vehicle operates a laser to project a known mark ontoits environment. When the mark illuminates the object, optics (e.g.,cameras) can visualize the mark and thus, with corresponding logic,determine the distance to the object. Accordingly, by causing itscamera(s) to process the optical image data after illuminating a portionof the object, the vehicle may determine the distance to the object. Forexample, the vehicle may determine that the object is sufficientlydistant that the vehicle does not need to alter the direction of itscourse.

Although projecting the laser's mark can be useful to the vehicle,powering the laser too often may unnecessarily drain the vehicle's powersupply. To avoid depleting the vehicle's power supply, the vehicleoperates the laser only when the vehicle cannot determine the distanceto an object using routine processing of image data from its cameras.

FIG. 1 depicts an exemplary vehicle 100 according to one embodiment ofthe present invention. The vehicle 100 has a processing unit 105 thatcontrols the operations of the vehicle's 100 components. In thisembodiment, the processing unit 105 is coupled to a steering component107 that positions the vehicle's treads 110 a, 110 b (collectively“110”), although in other embodiments, the steering component 107 iscoupled to any other type of component that enables the vehicle to move(e.g., wheels).

The processing unit 105 is coupled to at least two cameras 115 a, 115 b(referred to collectively as “115”) and a laser 120. In someembodiments, the cameras 115 are mounted on the vehicle 100, whereas inother embodiments, the cameras 115 are incorporated into the body of thevehicle 100. Likewise, the laser 120 may be mounted on the vehicle 100or incorporated therein.

The cameras 115 and laser 120 may face substantially the same direction.In some embodiments, the cameras 115 are positioned such that theirimage sensors are substantially parallel. Further, both cameras 115 maybe aligned with the front edge of the vehicle 100. The cameras 115 maybe sufficiently close together so that their image sensors capturesubstantially similar views. FIG. 2 depicts this overhead view of thevehicle 100, the cameras 115, and the fields of view 116 a and 116 b ofthe cameras 115. The views may be substantially similar if at least aportion of the views overlap. For example, the views may besubstantially similar if more than 60% of the view captured by onecamera 115 a is also present in the view captured by the other camera115 b, although other thresholds of overlap may be used. In someembodiments, the views may be substantially similar if an angle 118created by the fields of view of the cameras 115 exceeds a threshold.For example, if the fields of view of the cameras 115 create an angle118 greater than 45 degrees, the cameras' 115 views may be substantiallysimilar.

In some embodiments, the cameras 115 are positioned such that theirimage sensors are substantially parallel. For example, one camera 115 bmay be positioned behind the other camera 115 a along the body of thevehicle 100 so that one camera's field of view 116 b encompasses theother camera's field of view 116 a. FIG. 3 depicts this overhead view ofthe vehicle 100, the cameras 115 a and 115 b, and the fields of view 116a and 116 b of the cameras 115. In these embodiments, the cameras 115capture substantially the same field of view, although they are disposedat different distances from objects in the vehicle's 100 path.

In some embodiments, the cameras 115 are positioned such that theirimage sensors are angled towards one another. The cameras 115 may bespaced apart on the vehicle 100, and due to their angles, their fieldsof view intersect so that the cameras 115 captures substantially similarviews of the environment in front of the vehicle 100. FIG. 4 depictsthis overhead view of the vehicle 100, the cameras 115, and the fieldsof view 116 a and 116 b of the cameras 115.

The cameras 115 may include any kind of image sensor for capturing imagedata. For example, the cameras 115 may include charge coupled devices(CCD) for capturing image data. In another example, the cameras 115 mayinclude complementary metal-oxide-semiconductor (CMOS) active pixelsensors. The laser 120 may be any of a wide variety of different kindsof lasers that project a light ray. Exemplary lasers may include gaslasers, chemical lasers, excimer laser, solid state lasers, photoniccrystal lasers, or semiconductor lasers.

In operation, the cameras 115 capture image data and send the data tothe vehicle's processing unit 105. The processing unit 105 uses thisdata to attempt to identify one or more objects in the vehicle's 100path. The processing unit 105 may use any number of image processingalgorithms to determine whether the image data from the cameras 115contains one or more objects in the vehicle's 100 pathway, and whetherthe processing unit 105 can determine the distance between the vehicle100 and the object(s). In some situations, the presence of an objectand/or the ability to determine the distance to the object may beambiguous.

If either one is ambiguous, the processing unit 105 activates a laser120 to shine a discrete mark. The discrete mark presumably impinges uponan object in the vehicle's 100 path, consequently illuminating a portionof the object. The processing unit 105 therefore determines the distanceto the object based on the discrete mark; i.e., its cameras 115 locatethe mark, which enables the system to determine the distance to, orpresence of, the object. Conversely, if the presence of the object orthe ability to determine the distance to the object is not ambiguous,the processing unit 105 does not activate the laser 120, but determinesthe distance from the cameras' 115 image data.

In some embodiments, the processing unit 105 only processes a portion ofthe image data, e.g., the portion corresponding to the projected pathwayof the vehicle 100. For example, the processing unit 105 may processonly the image data within the central 60% of each camera's 115 view,thereby ignoring image data at the periphery of the views.

The laser 120 preferably projects the light ray as a discrete mark. Insome embodiments, the discrete mark is a dot, a square, a diamond, or anirregular shape. The discrete mark may have a curvative. The discretemark may have a straight edge. In some embodiments, the discrete markhas a solid interior, whereas in other embodiments, the discrete markhas a patterned interior. FIGS. 5-10 depict exemplary discrete marksthat the laser 120 could project.

Using a discrete mark also enables more accurate identification of thelaser by the cameras in a variety of light conditions that other laserswithout such a discrete mark or a pattern may not be able to handle. Forexample, it has been found that environments where the lighting isbright such as sunlight or where there may be reflections or otherconditions do not deter laser identification by cameras when the laserprojects such discrete marks.

In some embodiments, the vehicle 100 stores a template in memorycorresponding to the discrete mark. The template solely may include theshape of the discrete mark, whereas in other embodiments, the templateincludes grayscale values regarding the interior of the discrete mark.

The processing unit 105 may activate the laser 120 to project thediscrete mark for a predetermined period of time (e.g., 500 ms, 1 s). Insome embodiments, the processing unit 105 may operate the laser 120until the processing unit 105 identifies shapes corresponding to thelaser's 120 discrete mark in views for both cameras 105. For example,the processing unit 105 may detect edges in the image data and comparethe edges to the template shape. In another example, the processing unit105 may compare grayscale values in the image data with grayscale valuesin the template for the discrete mark. Once the processing unit 105identifies the marks in the image data, the processing unit 105deactivates the laser 120.

The processing unit 105 determines the distance to the object upon whichthe discrete mark impinges. The processing unit 105 may determine thedistance according to any number of algorithms, such as stereo visionalgorithms.

The vehicle 100 may be calibrated before being deployed. For example,the vehicle 100 may be placed in front of different objects at differentdistances. In these situations, the distance between the vehicle 100 andan object upon which the discrete mark impinges, the offset of thediscrete mark between images, and the angles are known. From these knownquantities, the factor may be calculated and stored, and thus used forfuture calculations of distance.

In some embodiments, the processing unit 105 determines the distance tothe object using image data from a single camera 115 a or 115 b. Theprocessing unit 105 may compute the distance as:distance to object (mm)=((focal length (mm)*real height of the discretemark (mm)*height of sensor (pixels))/(object height (pixels)*sensorheight (mm))

Use of a discrete laser mark also enables a human to be able to moreaccurately interact with the robot, particularly in the rare occasionwhen the robot gets hung up on an anomaly that cannot be resolved by thecurrent system. In such situations, a human can view the scene in personor perhaps via one or more of the cameras from the robot and see fromthe laser what particular item is causing confusion. Remote viewing ofthe cameras can take place on any type of handheld device, PC or othertype of visual interface device or the like. If desired, the laser canalso be activated to provide a message to a human rather than a merediscrete mark to help interact and communicate with the human. Thismessage, image, note or the like can take on a variety of forms asdesired.

In yet another embodiment, closing the loop between the laser and thecamera system to verify what each is identifying can be an importantfeature for a number of reasons. For example if the cameras identifysome type of anomaly, having the laser shine on the right anomaly thecameras have provides additional verification which provides anadditional speed, quality or safety control feature. This confirmationbetween the cameras and the laser comes into play in a number ofsituations including, for example, when say a pick and place or otherrobot may be using bar codes or other identifiers for location orpicking one or more items. When using just lasers for bar code typeidentification such lasers frequently need calibration and humaninteraction to continue operations. At least one embodiment of the robotenables the cameras to identify bar codes subject to laser confirmation.At least another embodiment of the robot enables the laser to identifybar codes subject to camera confirmation where the system sees andconfirms when the laser is pointing to the right bar code or otherobject. Of course, various other types of applications can beimplemented with the system described herein that in one way or anotherclose the loop or sync up the articles or the like sensed by the laserand cameras.

FIG. 11 shows a method of autonomously operating a vehicle in accordancewith illustrative embodiments of the invention. The method includesproviding at least two cameras in operable communication with thevehicle for providing substantially similar views relative to thevehicle. These cameras receive information relating to the views (step1101). The method also provides a laser in operable communication withthe vehicle for shining a single discrete mark on at least a portion ofthe views provided by the cameras (step 1103). Next, the methoddetermines whether the information received by the cameras is ambiguousregarding the views (step 1105). The method thus activates the laser onat least a portion of the views based on whether the informationreceived by the at least two cameras is ambiguous (step 1107). As notedabove, the laser shines a discrete mark on the object, thus making theobject visible to the cameras 115. Those cameras 115 thus are able tovisualize the object and, in conjunction with other logic, determine thedistance to, or presence of, the object.

In various embodiments, the processing unit 105 may determine thedistance using a conventional algorithm, not discussed herein.

It is understood that the present invention is not limited to theparticular components, analysis techniques, etc. described herein, asthese may vary. It is also to be understood that the terminology usedherein is used for the purpose of describing particular embodimentsonly, and is not intended to limit the scope of the present invention.It must be noted that as used herein, the singular forms “a,” “an,” and“the” include plural reference unless the context clearly dictatesotherwise. The invention described herein is intended to describe one ormore preferred embodiments for implementing the invention shown anddescribed in the accompanying figures.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meanings as commonly understood by one of ordinary skillin the art to which this invention belongs. Preferred methods, systemcomponents, and materials are described, although any methods andmaterials similar or equivalent to those described herein can be used inthe practice or testing of the present invention.

Many modifications and variations may be made in the techniques andstructures described and illustrated herein without departing from thespirit and scope of the present invention. Accordingly, the techniquesand structures described and illustrated herein should be understood tobe illustrative only and not limiting upon the scope of the presentinvention. The scope of the present invention is defined by the claims,which includes known equivalents and unforeseeable equivalents at thetime of filing of this application.

Although the description above contains many specific examples, theseshould not be construed as limiting the scope of the embodiments of thepresent disclosure but as merely providing illustrations of some of thepresently preferred embodiments of this disclosure. Thus, the scope ofthe embodiments of the disclosure should be determined by the appendedclaims and their legal equivalents, rather than by the examples given.

It will be appreciated by those skilled in the art that changes could bemade to the embodiments described above without departing from the broadinventive concept thereof. It is understood, therefore, that thisdisclosure is not limited to the particular embodiments disclosed, butit is intended to cover modifications within the spirit and scope of theembodiments of the present disclosure.

I claim:
 1. A method comprising: providing a stand-alone laser inoperable communication with a vehicle for selectively shining a singlediscrete mark on at least a portion of a view provided to a camera onthe vehicle, wherein a shape of the single discrete mark is determinedfrom a template stored in a memory of the vehicle; determining whetherinformation received by the camera is at least ambiguous regarding theview and therefore capable of more than one interpretation with regardto a direction of travel of the vehicle to yield an ambiguity; inresponse to the ambiguity, activating the stand-alone laser to projectthe single discrete mark into the view, wherein the single discrete markis projected in the shape determined from the template stored in thememory of the vehicle to yield a projected single discrete mark;detecting the projected single discrete mark within a correspondingportion of the view, the detecting based on analyzing objects containedthe view against expected shape information of the projected singlediscrete mark contained within the template; calculating a distance toan object upon which the projected single discrete mark impinges, thecalculating based on image data contained within a corresponding portionin which the projected single discrete mark was detected in the view, inorder to thereby resolve the ambiguity.
 2. The method of claim 1,further comprising: providing a vision system having two cameras inoperable communication with the vehicle for providing substantiallysimilar views relative to the vehicle, each of the two cameras receivingrespective information relating to the view and the view at least beingsubstantially in the direction of travel of the vehicle, the two camerasalone at least capable of determining distances between the vehicle andany objects in a path of the vehicle.
 3. The method of claim 2, whereinproviding the two cameras comprises providing overlapping views relativeto the vehicle.
 4. The method of claim 2, wherein providing the twocameras comprises providing views relative to the vehicle that overlapby at least a threshold percentage.
 5. The method of claim 2, whereinproviding the two cameras comprises providing views relative to thevehicle whose fields of view overlap by at least a threshold angle. 6.The method of claim 1, wherein the shape comprises a dot, a square, adiamond, or an irregular shape.
 7. The method of claim 1, wherein theshape comprises the single discrete mark with a curvature.
 8. The methodof claim 1, wherein the shape comprises the single discrete mark with astraight edge.
 9. The method of claim 1, wherein the shape comprises thesingle discrete mark having a solid interior.
 10. The method of claim 1,wherein determining whether the information is ambiguous comprises:determining whether a result of a formula when input based on theinformation in a substantially similar view is used in the formulaexceeds a predetermined threshold.
 11. The method of claim 1, whereindetermining whether the information is ambiguous comprises determiningwhether a distance to an object in the view can be ascertained.
 12. Themethod of claim 1, wherein determining whether the information isambiguous comprises determining whether a distance to an object in theview can be logically determined.
 13. The method of claim 1, whereinactivating the stand-alone laser based on whether the information isambiguous comprises activating the stand-alone laser only when theinformation is ambiguous.
 14. The method of claim 1, wherein activatingthe stand-alone laser based on whether the information is ambiguouscomprises not activating the stand-alone laser when the information isnot ambiguous.
 15. The method of claim 1, wherein activating thestand-alone laser comprises activating the stand-alone laser on at leasta portion of an object to illuminate the portion to the camera.
 16. Themethod of claim 15, further comprising using the camera to determine thedistance to an illuminated object.
 17. A vehicle comprising: a motor; aframe upon which the motor is configured; a camera; a memory; a controlsystem controlling the motor to move the vehicle; and a stand-alonelaser in operable communication with the control system, wherein thecontrol system stores instructions for controlling the stand-alone laserto perform operations comprising: selectively shining a single discretemark on at least a portion of a view provided to the camera on thevehicle, wherein a shape of the single discrete mark is determined froma template stored in the memory of the vehicle; determining whetherinformation received by the camera is at least ambiguous regarding theview and therefore capable of more than one interpretation with regardto a direction of travel of the vehicle to yield an ambiguity; inresponse to the ambiguity, activating the stand-alone laser to projectthe single discrete mark into the view, wherein the single discrete markis projected in the shape determined from the template stored in thememory of the vehicle to yield a projected single discrete mark;detecting the projected single discrete mark within a correspondingportion of the view, the detecting based on analyzing objects containedthe view against expected shape information of the projected singlediscrete mark contained within the template; and calculating a distanceto an object upon which the projected single discrete mark impinges, thecalculating based on image data contained within a corresponding portionin which the projected single discrete mark was detected in the view, inorder to thereby resolve the ambiguity.